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ABSTRACT 

Objectives: To test the performance of new variants 
of models to identify people at risk of an emergency 
hospital admission. We compared (1) the impact of 
using alternative data sources (hospital inpatient, A&E, 
outpatient and general practitioner (GP) electronic 
medical records) (2) the effects of local calibration on 
the performance of the models and (3) the choice of 
population denominators. 
Design: Multivariate logistic regressions using 
person-level data adding each data set sequentially 
to test value of additional variables and 
denominators. 

Setting: 5 Primary Care Trusts within England. 
Participants: 1 836 099 people aged 18-95 
registered with GPs on 31 July 2009. 
Main outcome measures: Models to predict 
hospital admission and readmission were compared in 
terms of the positive predictive value and sensitivity for 
various risk strata and with the receiver operating curve 
C statistic. 

Results: The addition of each data set showed 
moderate improvement in the number of patients 
identified with little or no loss of positive predictive 
value. However, even with inclusion of GP electronic 
medical record information, the algorithms identified 
only a small number of patients with no emergency 
hospital admissions in the previous 2 years. The model 
pooled across all sites performed almost as well as the 
models calibrated to local data from just one site. 
Using population denominators from GP registers led 
to better case finding. 

Conclusions: These models provide a basis for wider 
application in the National Health Service. Each of the 
models examined produces reasonably robust 
performance and offers some predictive value. 
The addition of more complex data adds some value, 
but we were unable to conclude that pooled models 
performed less well than those in individual sites. 
Choices about model should be linked to the 
intervention design. Characteristics of patients 
identified by the algorithms provide useful 
information in the design/costing of intervention 
strategies to improve care coordination/outcomes for 
these patients. 



ARTICLE SUMMARY 



Article focus 

■ The use of statistical models to predict risk of 
hospital admissions is increasingly used to pri 
oritise patients for preventive care. Models exist 
in several different forms and use a variety of 
input data sets. 

■ This paper compared the performance of a 
variety of models built using different data sets. 

Key messages 

■ The addition of more detailed data sets led to mod 
erate improvement in the number of patients iden 
tified with little or no loss of positive predictive 
value. 

■ The use of general practitioner registry data for 
the denominator proved to be of significant 
importance. By including all patients in an area, 
not just those with prior hospital use, improved 
rates of case finding were observed. 

■ Models calibrated to local data sets did not show 
consistent improvement over models built on 
pooled data. 

Strengths and limitations of this study 

■ The analysis is based on populations from only 
five areas in England; however, this is one of the 
largest UK populations (1.8 million people) used in 
the development of a publicly available risk tool. 

■ The success of a predictive model depends on 
many factors beyond the statistical performance 
of the model. 



INTRODUCTION 

There remains continuing interest in identi- 
fying patients at risk of future hospital admis- 
sions. Policies providing penalties 1 or 
non-payment 2 for hospital readmissions that 
put providers at risk for a share of total 
health expenditures have been developed in 
the USA and England. These create even 
stronger incentives to identify high-risk 
patients to target care coordination and man- 
agement strategies that may potentially 
reduce future inpatient expenditures. 
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Most predictive modelling approaches have used 
administrative data from claims in the USA or hospital 
data from hospital episode statistics or secondary uses 
services (SUS) in England. These data provide the infor- 
mation on prior utilisation and diagnostic history to 
develop predictive models for patients at risk of future 
hospitalisation. 3 Payor claims data in the USA provide 
rich information on care provided in hospitals, home 
care services and nursing home use, as well as detailed 
pharmacy prescription history. 4 In England, the most 
commonly used models (such as in the now outdated 
patient at risk of readmission (PARR) algorithm) 5 6 are 
based on hospital admissions data (including day case 
use and regular attendances) with some use of accident 
and emergency (A&E) and outpatient attendance data 
as well. 7 

While some predictive modelling efforts in the UK 
have included information from general practitioner 
(GP) electronic medical records (EMRs), 8 9 using such 
data presents a number of challenges. These include 
obtaining permissions for access to EMRs for large popu- 
lations, linking the records to hospital data and use of 
Read codes 10 to develop GP variables. Data from EMRs 
include additional elements not available in the hospital 
data sets such as test results (eg, blood pressure, glycated 
haemoglobin (HbAlc) levels), diagnostic history for 
patients without recent inpatient admissions, prescrip- 
tion history, GP contact patterns (GP visits and tele- 
phone contacts) and other personal health markers (eg, 
body mass index, smoking status). These additional data 
elements have the potential to add power to predictive 
modelling efforts, especially for patients with no or 
lower levels of recent inpatient use. 

Despite the challenges, a number of initiatives around 
the UK are demonstrating population-wide access to 
EMR data. A common application is in the use of 
models that assist local clinical commissioning groups in 
identifying high-risk patients. 

Though the choice of data sets for a predictive model 
makes a big difference to the investment required to 
run these models — at least initially — no studies have 
looked at the marginal value of different data sets. In 
this analysis, we examine the added value of including 
data on A&E and outpatient visits (which are readily 
available) to predictive modelling efforts using hospital 
inpatient data alone. We also assess the marginal effect 
of adding GP EMR information to help identify patients 
at risk of future hospital admissions. Most of the existing 
models in use were developed using logistic regression 
techniques and we used this standard approach through- 
out this paper. We recognise that different modelling 
methods may yield different results, but in this analysis 
we were concerned with the impact of changes in the 
underlying data sets. Such models will always be limited 
by the scope and quality of data available, the ways data 
are grouped and classified and the ways that users can 
assess up-to-date information. Despite these problems, 
these models have become commonly used tools. In 



addition to the depth of data used, there is also a ques- 
tion about how generalisable models are across different 
sites. In many settings, models are recalibrated on local 
data sets. Yet there is little systematic analysis of the value 
that this step adds and whether models built on data 
from one site outperform those built on a larger sample 
of pooled data. We therefore explore whether there is a 
need for development of individual site predictive 
models, or whether models developed from multiple 
sites can be applied effectively at a new individual site. 

METHODS 

We conducted analyses separately for five Primary Care 
Trust (PCT) areas in England (Newham, Cornwall, 
Kent, Croydon, Redbridge; total adult population 
ranging from 209 661 to 693 089). Results are reported 
for the individual sites and as combined/pooled results 
(total population 1 836 099). Hospital data were 
extracted from the SUS 11 system which contained 
records of all hospital events (inpatient admissions, out- 
patient appointments and A&E visits) for the PCTs' 
registered populations between 1 August 2007 and 
30 September 2010. The PCTs also extracted data from 
GP systems in two forms. First, as a register of the local 
adult population from 1 August 2007 to 31 July 2009, 
and second, in the form of data sets recording details of 
GP consultations over the same time period. 

Personally identifiable information was stripped out 
before any data were passed to the research team. 
Individuals' NHS numbers (the personal identifiers) 
were concatenated with a pass code chosen by each of 
the five PCT areas (and unknown to the research team), 
and these were pseudonymised at source using the 
secure hash algorithm SHA-256. 12 This allowed for 
linkage between the hospital and the general practice 
data from each area, while preserving the individuals' 
anonymity. 

A series of variables were created from each data set 
that were believed to be potentially predictive of an 
unplanned (emergency) hospital admission in the last 
12 months of the study period. These variables captured 
resource use, utilisation patterns, diagnostic history, test 
results and prescription history in the 2 years prior to 
the predictive period. They were created for all indivi- 
duals aged 18+ years and registered with a GP in one of 
the five areas on 31 July 2009. To account for the 
expected time required to obtain and process the hos- 
pital and GP EMR data, we included a 2-month lag in 
our analyses, with data from 1 August 2007 to 31 July 
2009 used to predict emergency admissions during the 
period 1 October 2009 to 30 September 2010. 

Patient age and gender were obtained from the GP 
register. Patient area of residence was not available, and 
therefore GP practice attributed index of multiple 
deprivation (2007) was used as an area deprivation 
measure. The number of months the patient was regis- 
tered with the PCT in the preperiod was calculated and 
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included in the regression. Hospital inpatient data were 
used to capture utilisation in the 0-90, 91-181, 180-365 
and 366-730 days prior to the lag period. The number 
of emergency and elective admissions for these periods 
was included and dichotomous variables for any day case 
or regular attendance use were created. 

A broad range of diagnostic variables were developed 
using primary and secondary diagnosis fields and a 
Charlson Comorbidity Index 13 was calculated for each 
patient and included in the model. 

A&E data were used to determine A&E visit rates for 
various intervals in the preperiod, both total visits and 
unplanned follow-up visits. A&E diagnostic information 
was not reliably reported across the five sites and was not 
included, although X-ray use was included. Outpatient 
data provided variables on outpatient visit rates for 
various intervals, as well as missed appointment rates 
and the number of different specialty types consulted. 
Diagnostic information in outpatient data was missing in 
more than 95% of cases and was not included. 

GP EMR data were used to create proxy visit rates 
(these may include both actual GP visits, in addition to 
other events documented in a person's records) for 
various intervals and to capture any increase in visit 
rates at the end of the preperiod that may reflect 
increased morbidity in a patient. EMR Read codes 
(CTV3 version) were used to obtain test results (blood 
pressure, blood serum levels, HbAlc levels, etc), body 
mass index, smoking history, prescription history 
(number and type) and a range of diagnostic variables 
during the preperiod. 

Variables from each data set (inpatient (including day 
case and regular attenders), A&E, outpatient and GP 
EMRs) were added and modelled sequentially using 
standard logistic regression in SPSS V.20. Emergency 
admission in the next 12 months was used as the 
dependent variable, producing a risk score ranging from 
0 to 100. Separate models were developed for each PCT 
area, and analysis was limited to patients aged 18-95 
who were on the GP register in the area. Over-fitting was 
tested using a split sample approach, with only minor 
differences observed in positive predictive values (PPV), 
sensitivity and specificity. 

The findings provided here include individual site 
results and results combined across the five sites. We also 
created five additional predictive models (referred to 
below as the 'four-site regression models'), each one 
combining data from four sites and applying coefficients 
to the fifth remaining site. With this, we could compare 
results with individual site predictive models to help 
assess the value of local model development. 

The full list of more than 300 potential variables was 
ultimately reduced to 88 by exclusion of variables with 
low volumes and low significance levels across the sites. 
The 88 variables ultimately included in the model (and 
regression coefficients) may be found in online supple- 
mentary appendices B and D, and a full listing of the 
variables considered for inclusion and detailed 



specification of each variable are available at http:// 
www.nuffieldtrust.org.uk/. 

Cost variables were examined, with secondary care 
activity costed according to the method used in develop- 
ment of the person-based formula for allocating com- 
missioning funds to general practices in England. 14 
Ultimately, these were not included in the predictive 
models because of concerns about difficulties in con- 
structing these variables by possible future users; 
however, costs are included in descriptive findings to 
help in the design of intervention strategies. 

Predictive modelling performance is typically docu- 
mented reporting PPV and sensitivity at the risk score 
threshold of 50. However, because interventions may be 
targeted at patients with higher or lower risk scores and 
intervention strategies may be calibrated differently 
depending on the risk level and characteristics of 
patients at various risk score levels, we report PPV sensi- 
tivity at 20 risk score cut-off points (vigintiles) and 
provide detailed patient characteristics at risk score 
thresholds of 50 and 30 to facilitate intervention design. 

RESULTS 

Pooled individual site results 

There were 1 836 099 people aged 18 and over who 
were registered with a GP practice on 31 July 2009. 
Table 1 shows the combined results of individual site 
regressions including the number of patients correctly 
identified, PPV and sensitivity for four models: 

1. IP based on hospital inpatient data only (including 
day cases and regular attendances) ; 

2. IPAE using inpatient and A&E data; 

3. IPAEOP using inpatient, A&E and outpatient data; 

4. IPAEOPGP using inpatient, A&E, outpatient data and 
GP EMR. 

At the traditional risk score threshold level of 50, all four 
models perform respectably in terms of PPV (ranging 
from 0.523 to 0.538), but sensitivity remains quite low 
across all models (0.049-0.060). Lowering the threshold to 
30 increases sensitivity somewhat with a concomitant 
reduction in PPV (ranging from 0.417 to 0.422). 
The receiver operator characteristic area under the curve 
(C statistic) improved with the addition of each data set, 
increasing from 0.731 with the inpatient-only model to 
0.780 with the full model. 

Of particular note is the finding that the addition of 
each data set added power, that is, correctly identified 
more patients with an admission in the next 12 months, 
with only a minor reduction in PPV. At a risk threshold 
of 50, the addition of A&E data resulted in an increase 
of 400 (8.6%) correctly flagged patients, with no loss in 
PPV. The inclusion of outpatient data added a more 
modest 2.9%, but with a slight loss in PPV (0.531 to 
0.523). The addition of GP EMR data added an add- 
itional 9.6% of patients, while actually increasing the 
accuracy of the model (PPV increasing from 0.523 to 
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Table 1 Model performance, four models: IP, IPAE, IPAEOP, IPAEOPGP 



Risk score 



IP data 



IP+AE data 



IP+AE+OP data 



IP+AE+OP+GP Data 



threshold 


True-positive 


PPV 


Sensitivity 


True-positive 


PPV 


Sensitivity 


True-positive 


PPV 


Sensitivity 


True-positive 


PPV 


1 


QA CZQO 


n nco 


■i nnn 
1 .UUU 


QA CZQO 

y4 oy^i 


n nco 


■i nnn 
1 .UUU 


QA COO 

y4 by^i 


n nco 


a nnn 
1 .UUU 


QA CZQO 

y4 by^ 


n nco 


C 

D 


o4 4oU 


n a 

U.l ^lb 


n C7c 

U.O/ O 


ro A A ~7 
DO 11/ 


n A OQ 

U.l do 


n CQQ 

u.oyo 


ro a QQ 

DO 4oo 


U.l ol 


n cqc 
U.oyb 


OH AQQ 

01 4yo 


n A QQ 
U.l 00 


a n 
1 U 


oo Uoo 


u.^i y 


n q/iq 
U.o4y 


Q/i i no 
o4 1 U^i 


n ooi 
U.^l 


n Qcn 
U.obU 


nc nQQ 

00 Uoo 


n ooq 


n Qvn 
U.o/ U 


QQ QQC 

oy yob 


n oon 
V.ddU 


A K 
ID 


on QQQ 


U.^ioo 


n o/io 


^lO 1 DO 


n OQQ 


n o/ik 


OA OCZA 

^4 ^ibl 


n oon 

u.^yu 


n occ 


OQ CQ7 

do by/ 


n OQQ 

U.^100 


on 


lb 1 ol 


U.o4b 


U.I / I 


H O DH C 

i b yi O 


n o/i~7 
U.o4/ 


n h ~7n 

u.i /y 


i / / 1 y 


n Q/i a 
U.o44 


n H Q"7 

U.l 0/ 


0 a onn 
dl bUl 


n QQQ 

U.000 


OK 




n QQC 

U.ooO 


U.l o4 


HQ H QO 

lo loV 


U.oob 


n i QQ 

u.i oy 


H Q ~7CA 

lo /o4 


n QQQ 

U.ooo 


r\ A AC 

U.l 4b 


A d CZ~70 

1 b old. 


n Q"7Q 

U.o/ 0 


30 


10 061 


0.421 


0.106 


10 555 


0.422 


0.111 


11 010 


0.419 


0.116 


13 196 


0.417 


35 


8130 


0.449 


0.086 


8600 


0.450 


0.091 


8986 


0.448 


0.095 


10516 


0.450 


40 


6700 


0.477 


0.071 


7139 


0.478 


0.075 


7421 


0.476 


0.078 


8494 


0.479 


45 


5535 


0.501 


0.058 


5976 


0.504 


0.063 


6167 


0.499 


0.065 


6921 


0.510 


50 


4627 


0.529 


0.049 


5027 


0.531 


0.053 


5172 


0.523 


0.055 


5669 


0.538 


55 


3862 


0.551 


0.041 


4222 


0.551 


0.045 


4359 


0.543 


0.046 


4581 


0.562 


60 


3239 


0.574 


0.034 


3555 


0.569 


0.038 


3658 


0.567 


0.039 


3735 
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65 


2711 
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0.029 
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0.032 


3041 


0.587 


0.032 


3034 
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70 


2245 
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0.024 


2481 
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0.026 
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0.027 
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75 
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0.019 
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0.022 


2064 


0.631 


0.022 


1921 
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80 


1418 
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0.015 


1662 


0.656 


0.018 


1646 
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0.017 


1478 
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85 


1064 


0.679 


0.011 


1293 


0.674 


0.014 


1276 


0.679 


0.013 


1114 


0.711 


90 


769 


0.710 


0.008 


932 


0.688 


0.010 


935 


0.702 


0.010 


754 


0.738 


95 


478 


0.748 


0.005 


592 


0.725 


0.006 


586 


0.728 


0.006 


437 


0.771 


Top 1% 


8214 


0.447 


0.087 


8353 


0.455 


0.088 


8410 


0.458 


0.089 


8722 


0.475 


Top 5% 


24 873 


0.271 


0.263 


25 355 


0.276 


0.268 


25 712 


0.280 


0.272 


26 991 


0.294 
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0.731 
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Five site individual runs combined. 

PPV, positive predictive values; ROC, receiver operator characteristic. 
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0.538). The added power of the A&E data set is less sub- 
stantial at a risk score threshold of 30 (4.9%), but out- 
patient and GP EMR data sets had larger increases in 
correctly identified patients (4.3% and 19.9%). 

There were also important differences between the 
models in terms of the characteristics of patients identi- 
fied as high risk. For example, at a risk score cut-off 
of 50, patients identified using inpatient data alone had 
high prior emergency inpatient utilisation rates with 
2.62 admissions in the previous year compared to 2.43 
when A&E data were added; 2.34 with the addition of an 
outpatient and 2.20 with the addition of GP EMR data 
(see table 2). 

The inclusion of additional data sets also led to a 
reduction in the observed morbidity level of patients at 
the 50 threshold, with lower numbers of long-term con- 
ditions, fewer patients with multiple long-term condi- 
tions, lower Charlson Comorbidity Index scores, less 
history of alcohol abuse and mental illness and lower 
emergency inpatient costs in the years prior to the pre- 
dictive period. Similar, but less substantial, differences 
were observed at the risk score threshold of 30. The add- 
ition of the A&E data set resulted in higher rates of A&E 
visits in the preperiod among patients identified at both 
risk score cut-off levels, and the addition of outpatient 
data resulted in higher outpatient visit and missed visit 
rates among identified patients. 

These findings suggest that the inclusion of additional 
data sets added some predictive power and generally 
tended to find additional patients who were less severely 
ill (more severely ill patients tended to remain high 
risk). Thus, they potentially offer an opportunity for 
intervention at earlier stages in the progression of a 
patient's condition. However, the number of patients 
identified with no prior emergency inpatient utilisation 
in the previous 2 years was relatively small across all 
models. At a risk score threshold of 50, only 0.3% of 
patients correctly identified by the inpatient-only model 
had no prior emergency admissions in the previous 
2 years, and increased only modestly 3.2% in the full 
model (table 3). At a risk threshold of 30, the rates were 
higher, but only reached 12.4% for the full model. 

Individual site and 'four-site regression' model results 

Overall, the performance of the models was similar at 
the individual site level. Only modest differences were 
found in PPV levels and sensitivity across the sites. For 
runs using non-GP data only (IPAEOP), at a risk score 
threshold of 50, PPVs ranged from 0.512 to 0.552 and 
sensitivity ranged from 0.047 to 0.071. For the model 
including GP EMRs, PPVs ranged from 0.521 to 0.566 
and sensitivity from 0.053 to 0.073 (see online supple- 
mentary appendix A). There was some variation in the 
magnitude of regression coefficients between sites, but 
in general the coefficients were comparable for the 
models based on the non-GP data model (IPAEOP) (see 
online supplementary appendix B). For the model 
including variables from GP EMRs (IPAEOPGP), the 



level of variation in regression coefficients (size and dir- 
ection) was somewhat greater for those variables derived 
from GP data. We observed substantial differences in fre- 
quency of reporting of Read codes across sites, which no 
doubt contributed to this variation. The level of signifi- 
cance of individual variables also varied across sites (see 
online supplementary appendix C), but most variables 
were consistently strongly significant across all sites, espe- 
cially variables involving prior emergency inpatient 
admissions. Again, higher levels of variation in levels of 
significance were observed for the GP variables derived 
from Read codes. 

We compared the results for these individual site 
models to that of a pooled model combining data from 
four of the sites and applied coefficients to the remain- 
ing individual site. We generally found only small differ- 
ences in predictive accuracy (PPV) between these two 
approaches (table 4); however, the individual site 
models identified a greater number of true positives. For 
example, in Cornwall at a risk score cut-off of 50, the 
individual site model using hospital data correctly identi- 
fied 1041 patients while the pooled model identified 
only 754 patients. In Newham, however, the four-site 
model was more powerful, correctly identifying 858 
patients compared to 734 patients for the individual site 
approach. In both cases (and in general across all sites), 
the model identifying larger numbers of true positives 
had a somewhat lower PPV, suggesting that improved 
case finding volume came at the expense of predictive 
accuracy. 

Testing alternative population denominators 

Models built using inpatient data only (IP) were also 
built for just the subset of patients who had some 
inpatient care in the previous 2 years (to reflect typical 
predictive modelling efforts that may have been con- 
ducted without access to GP registry information), as 
well as for the group who had had an emergency admis- 
sion in the previous year (to replicate analyses con- 
ducted by PARR users). 

Combining the results from the five sites at a risk 
score threshold of 50, models using the full GP register 
correctly identified 4627 patients compared with 3572 
patients in runs restricted to patients with prior inpatient 
care and 3060 to runs limited to patients with an emer- 
gency admission in the previous year. This substantial 
increase in case finding was obtained with only moder- 
ate loss in PPV (0.529 GP list, 0.559 previous inpatient 
and 0.589 emergency admissions in the last year). 
Similar results were also found for all hospital data 
models (IPOPAE, though with any hospital use in the 
previous 2 years, rather than just any inpatient use). 

Using the full GP registry population did not result in 
finding substantial numbers of patients with no emer- 
gency admissions in the previous 2 years, but the 
increased numbers of patients identified included more 
patients with less prior use and lower levels of morbidity. 
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Table 2 Patient characteristics by risk score threshold, four models: IP, IPAE, IPAEOP, IPAEOPGP 




Risk score 50+ 




Risk score 30+ 








IP 


IPAE 


in A r— 

IPAEOP 


IPAEOPGP 


IP 


IPAE 


IPAEOP 


IPAEOPGP 




Data 


Data 


Data 


Data 


Data 


Data 


Data 


Data 


Number of patients 


8743 


9473 


9892 


10 545 


23 912 


25 021 


26 304 


31 653 


Patients with admission next 12 months 


4627 


5027 


5172 


5669 


10 062 


10 554 


1 1 01 1 


13 196 


Mean age 


73.9 


72.3 


71.2 


72.2 


75.3 


74.0 


73.4 


73.9 


18-39 


6.2% 


8.4% 


8.7% 


7.8% 


5.1% 


6.8% 


7.0% 


6.4% 


40-54 


8.7% 


9.6% 


10.9% 


1 0.6% 


7.0% 


7.8% 


8.6% 


8.5% 


55-64 


6.9% 


7.4% 


7.8% 


8.1% 


6.4% 


6.4% 


7.0% 


7.3% 


65-74 


14.4% 


13.9% 


15.0% 


1 3.5% 


13.9% 


13.8% 


14.4% 


13.7% 


75-84 


30.5% 


28.9% 


28.2% 


27.4% 


32.1% 


31.1% 


30.3% 


29.6% 


85+ 


33.3% 


31.8% 


29.3% 


32.6% 


35.5% 


34.1% 


32.7% 


34.4% 


Female 


55.2% 


55.2% 


54.9% 


54.7% 


56.9% 


57.1% 


56.6% 


56.5% 


Practice IMD 


24.6 


24.9 


24.8 


24.3 


24.2 


24.3 


24.3 


23.8 


Ischaemic heart disease 


36.2% 


34.2% 


33.2% 


32.1% 


29.8% 


28.4% 


27.8% 


25.2% 


Angina 


21.7% 


20.5% 


19.5% 


1 9.3% 


17.3% 


16.4% 


15.7% 


1 4.5% 


Hypertension 


64.5% 


61.2% 


59.9% 


59.0% 


59.4% 


56.7% 


55.1% 


50.9% 


CHF 


19.2% 


17.8% 


16.9% 


15.9% 


14.0% 


13.2% 


12.5% 


10.7% 


CVD 


21.7% 


20.1% 


19.3% 


18.1% 


16.9% 


15.8% 


15.2% 


13.3% 


COPD 


23.4% 


21.4% 


20.6% 


19.1% 


17.2% 


16.3% 


15.7% 


13.4% 


Asthma 


21.3% 


20.9% 


20.1% 


18.1% 


17.4% 


16.6% 


16.1% 


13.8% 


Diabetes 


34.1% 


32.4% 


31 .6% 


28.8% 


30.1% 


28.8% 


27.6% 


23.6% 


Renal failure 


16.5% 


14.6% 


14.2% 


1 3.3% 


1 1 .2% 


10.4% 


10.0% 


8.5% 


Any long-term conditions 


89.8% 


87.0% 


85.8% 


oc no/ 


86.6% 


83.6% 


OH AO/ 

ol .4% 


"7C OO/ 

75. dvo 


Any cancer 


15.4% 


13.8% 


13.4% 


HO /IO/ 

1^.4% 


13.3% 


12.7% 


-\ o OO/ 

\d. d /o 


h r\ co/ 
10.5% 


Alcohol misuse 


10.0% 


10.0% 


9.6% 


Q HO/ 


7.0% 


6.7% 


a. AO/ 


C QO/ 
D.O/O 


Mental illness 


32.0% 


30.6% 


28.9% 


07 00/ 
dl .d /o 


23.5% 


22.3% 


OH QO/ 

dl .O /O 


H Q 00/ 

1 o.D /o 


Number of long-term conditions 


2.67 


2.51 


2.43 


2.31 


2.20 


2.10 


2.02 


1 .79 


Charlson Index 


3.96 


3.69 


3.55 


o oo 
o.do 


3.09 


2.95 


O OO 

d.od 


O A O 

dAo 


Emergency admission 1 year previously 


2.62 


2.43 


2.34 


o on 
2.20 


1.64 


1.57 


H CO 

1 .50 


H OO 

1 .29 


Emergency admission 2 years previously 


1.78 


1.67 


1.61 


1 .50 


1.15 


1.10 


H OC 

1 .05 


C\ O H 

o.yi 


No emergency admission in the previous 


0.6% 


1 .9% 


3.3% 


4.3% 


4.2% 


6.5% 


O 00/ 

9.2% 


H O 00/ 

16.2% 


2 years 


















Elective admission 1 year previously 


0.32 


0.29 


0.31 


O oo 


0.27 


0.26 


O 0"7 

0.^7 


O O A 

0.^4 


Elective admission 2 years previously 


0.24 


0.23 


0.26 




0.20 


0.20 


c\ oo 
KJ.dd 


C\ HO 


Any day case 1 year previously 


31.9% 


30.8% 


31 .6% 


28.7% 


31.5% 


30.7% 


30.4% 


26.9% 


Any day case 2 years previously 


28.9% 


27.6% 


28.0% 


OC "70/ 


27.8% 


26.9% 


OO "70/ 


OO "70/ 

^o.7% 


Emergency admission cost 1 year previously 


£4500 


£4073 


£3920 


£3688 


£2893 


£2709 


£2604 


£2231 


Emergency admission cost 2 years previously 


£2932 


£2675 


£2583 


T-d*\dd 


£1962 


£1822 


PH 7C7 
TL \ lOl 


OH COH 
TL\ Od\ 


AE visits 1 year previously 


2.90 


3.59 


3.45 


3.17 


1.83 


2.26 


2.17 


1 .86 


AE visits 2 years previously 


1.90 


2.40 


2.31 


O HO 

d.\ U 


1.23 


1.52 


H AG. 

\ .4b 


H OC 

\ .do 


OP visits 1 year previously 


7.27 


6.94 


9.65 


8.23 


5.65 


5.63 


7.39 


6.16 


OP visits 2 years previously 


4.30 


4.10 


5.92 


/- OO 

5.08 


3.39 


3.38 


4.54 


O O H 

3.81 


OP visits missed 1 year previously 


0.47 


0.48 


0.74 


0.65 


0.35 


0.36 


0.53 


0.44 


OP visits missed 2 years previously 


0.49 


0.48 


0.71 


0.61 


0.33 


0.34 


0.48 


0.40 


GP visits 1 year previously 


42.9 


42.7 


43.3 


52.5 


38.5 


38.4 


38.8 


45.4 


GP visits 2 years previously 


35.5 


35.2 


35.7 


42.5 


32.4 


32.1 


32.5 


37.7 


Any high-risk BNF codes 


73.9% 


71 .6% 


72.2% 


o a no/ 
o4.U% 


69.3% 


67.5% 


CO HO/ 


7H OO/ 

7y.o% 


Number of high-risk BNF codes 


1.94 


1.85 


1.88 


2.20 


1.64 


1.59 


1 .61 


1 .84 


High-blood pressure 


9.0% 


9.0% 


9.0% 


O 00/ 

y.o% 


10.0% 


10.0% 


H O OO/ 

10.0% 


o oo/ 

y.o% 


Smoker 


18.0% 


19.0% 


19.0% 


23.0% 


16.0% 


16.0% 


16.0% 


20.0% 


□ R /II OO . 

BMI 30+ 


16.0% 


17.0% 


17.0% 


1 7.0% 


15.0% 


16.0% 


16.0% 


18.0% 


HDA1C >1U 


5.0% 


5.0% 


5.0% 


-7 OO/ 

7.0% 


5.0% 


5.0% 


a r\o/ 
4.0% 


a oo/ 


Any Emergency admission in the next 


52.9% 


53.1% 


52.3% 


53.8% 


42.1% 


42.2% 


41.9% 


41 .7% 


12 months 


















Num Emergency admission in the next 


1.34 


1.33 


1.29 


1.31 


0.89 


0.89 


0.87 


0.84 


12 months 


















0 Emergency admission in the next 12 months 


47.1% 


46.9% 


47.7% 


46.2% 


57.9% 


57.8% 


58.1% 


58.3% 


1 Emergency admission in the next 12 months 


23.0% 


23.0% 


23.2% 


23.9% 


21.7% 


21.9% 


21.7% 


22.3% 


















Continued 
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Table 2 Continued 




Risk score 50+ 




Risk score 30+ 








IP IPAE IPAEOP 


IPAEOPGP 


IP 


IPAE 


IPAEOP 


IPAEOPGP 




Data Data Data 


Data 


Data 


Data 


Data 


Data 


2 EnriGrgGncy admission in the next 12 months 


i O CO/ H O CO/ H O QO/ 

\£..D/o \d.O/o \d.O/o 


H Q 00/ 


10.0% 


10.0% 


n n no/ 
1 U.U /o 


Q QO/ 

y.y /o 


3 Emergency admission in the next 12 months 


7.3% 7.3% 7.0% 


7 ho/ 

/ .1 To 


4.8% 


4.9% 


A QO/ 
4.O70 


A CO/ 

4.b% 


4+ Emergency admission in the next 


10.2% 10.2% 9.8% 


9.7% 


5.5% 


5.5% 


5.3% 


4.9% 


12 months 














Emergency admission cost in the next 


£2358 £2266 £2199 


£2270 


£1608 


£1575 


£1546 


£1507 


12 months 














AE visits in the next 12 months 


1.88 2.11 2.04 


2.04 


1.24 


1.37 


1.36 


1.29 


Four models: IP, IPAE, IPAEOP, IPAEOPGP. Five site individual runs combined. 












AE, accident and emergency; BMI, body mass index; BNF, British National Formulary; CHF, Congestive heart failure; COPD, chronic 


obstructive pulmonary disease; CVD, cardiovascular disease; GP, general practitioner; HbA1c, glycated haemoglobin; IMD, Index of Multiple 


Deprivation. 















For a profile of patients identified using these alternative 
denominators, see http://www.nuffleldtrust.org.uk/. 

DISCUSSION 

This analysis has looked at the performance of new var- 
iants of predictive models for case finding. These 
models are intended to update and improve upon the 
established combined predictive model-like 15 and PARR 
models 5 widely used in the NHS. 

Each of the models examined produced reasonably 
robust performance, by some measures better or at least 
comparable to similar prior models. 9 At a risk threshold 
of 50, patients identified by the models had PPVs 
ranging from 0.523 to 0.538. While the percentage of all 
patients with future admissions identified was relatively 
low (sensitivity 0.049-0.060), lowering the risk threshold 
allows the identification of more patients with relatively 
small loss in PPV (eg, at a risk threshold of 30, the full 
model identified 14% of future admissions with a PPV of 
0.417). Users of predictive modelling algorithms have 
obvious trade-offs between maximising the number of 
patients identified and predictive accuracy. Lower risk 
score thresholds will find more patients, but these 
patients are increasingly less likely to have future 
admissions. 

The implications for intervention design are import- 
ant. Patients at lower risk thresholds have less prior 
inpatient use and lower morbidity, so an intervention 
here might be calibrated to be less intensive. But 
because the models are less accurate at lower risk scores, 
the amount that can be spent on an intervention is also 



reduced if you wish to achieve financial break-even (ie, 
where the cost of intervention is offset by cost savings 
from reduction in future admissions) . As documented in 
table 2, at a risk score threshold of 50, the rate of future 
admission for patients identified by the full model 
(IPAEOPGP data) was 1.31 admissions per year with an 
associated cost of £2270. If there were a 10% reduction 
in future admissions, £227 could be spent on an inter- 
vention to improve care coordination and still achieve 
break even. However, at a lower risk threshold of 30, the 
lower rates of future admissions and costs means that 
lower intervention expenditures are required to achieve 
break even (£151 with a 10% reduction in future admis- 
sions). A detailed business case analysis with mean emer- 
gency inpatient costs in the next 12 months within each 
risk vigintile level is available via http://www 
nuffieldtrust.org.uk/ 

These data also provide other information that may be 
useful in the development of intervention strategies. As 
shown in table 2, patients identified by the models have 
extremely high rates of chronic disease (85-90% with 
long-term conditions at risk threshold of 50), often with 
multiple long-term conditions and high Charlson 
Comorbidity Index levels, indicating serious medical 
needs. However, these patients already have high use of 
outpatient care and very high GP visit rates. This sug- 
gests that simple access to ambulatory care is not the 
issue, but prevention needs to look at care coordination 
and management of complex problems and at the 
ability of patients and their families to manage chronic 
illness. High-risk patients identified by the models also 
have relatively high rates of mental illness (27-32% at 



Table 3 Proportion of patients correctly identified, who had no emergency admissions in the previous 2 years 


Prediction threshold 


IP data (%) 


IPAE data (%) 


IPAEOP data (%) 


IPAEOPGP data (%) 


Risk score 50+ 


0.3 


1.2 


2.3 


3.2 


Risk score 30+ 


2.7 


4.4 


6.3 


12.4 


Top 1% 


1.5 


2.9 


4.2 


6.5 


Top 5% 


25.9 


26.4 


26.7 


30.8 


Four models: IP, IPAE, IPAEOP, IPAEOPGP. Five site individual runs combined. 
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Table 4 Individual site and four-site regression models 




IPOPAE 








IPOPAEGP 








Individual-site 
regression 




Four-site regression 


Individual-site 
regression 




Four-site regression 




True-positives 


PPV 


True-positives 


PPV 


True-positives 


PPV 


True-positives 


PPV 


Newham 


















Risk score 50+ 


734 


0.552 


858 


0.517 


768 


0.566 


835 


0.523 


Risk score 30+ 


1409 


0.450 


1564 


0.414 


1570 


0.439 


1798 


0.409 


Cornwall 


















Risk score 50+ 


1041 


0.520 


754 


0.548 


1 176 


0.545 


952 


0.556 


Risk score 30+ 


2439 


0.406 


1970 


0.426 


3032 


0.410 


2746 


0.411 


fxeni 


















Risk score 50+ 


1565 


0.513 


1387 


0.519 


1736 


0.521 


1873 


0.493 


Risk score 30+ 


QQ70 


n ac\a 
U.4U1 


3067 


0.403 




n QQ7 


4432 


0.369 


Croydon 


















Risk score 50+ 


1089 


0.528 


1192 


0.523 


1182 


0.550 


1230 


0.537 


Risk score 30+ 


2134 


0.444 


2258 


0.424 


2610 


0.442 


2502 


0.437 


Redbridge 


















Risk score 50+ 


743 


0.512 


863 


0.495 


807 


0.522 


607 


0.519 


Risk score 30+ 


1656 


0.420 


1693 


0.415 


1905 


0.423 


1390 


0.436 


Case finding and predictive accuracy. 



risk threshold of 50) and moderate levels of alcohol 
abuse, factors that are likely to complicate any interven- 
tion strategy. 

It is also important to note the limitations of these 
data in helping frame the design of any intervention 
strategy. Other studies have documented that high-risk 
patients often have important characteristics related to 
care needs and patient capacity not captured by adminis- 
trative data and EMRs. For example, interviews with 
high-risk patients and their families have documented 
high levels of social isolation for many, as well as precar- 
ious housing status. 16 These non-medical factors are 
likely to have significant impact on health status and util- 
isation patterns. Moreover, not much is known about 
how/whether care coordination and management has 
actually failed for these patients. Are these high-risk 
patients just very sick patients whose hospitalisations are 
largely not preventable/avoidable, 17 or has the care 
delivery system failed in some important dimensions that 
can be corrected with improved care coordination and 
management? These data cannot answer this very critical 
question, and it is clear that the field would benefit 
from further study that examined the circumstances of 
patients identified as high risk by predictive modelling 
algorithms to sort out more clearly the factors contribut- 
ing to high rates of emergency admission. 

This study does document the value of incorporating 
data sets beyond inpatient records. The addition of A&E 
and outpatient records resulted in the identification of 
more high-risk patients with little or no loss of predictive 
accuracy. These data sets are readily available and have 
standardised reporting formats that facilitate analysis. 
While the absence of useful diagnostic information in 
these data sets is a limiting factor, the improvement in 



case finding and usefulness in descriptive profiling of 
high-risk patients to help in intervention design (eg, 
high rates of A&E use rates, high rates of missed out- 
patient appointments) suggests that their inclusion is 
clearly merited. 

The use of GP EMRs presents significant challenges. 
While the lack of access to these data is unlikely to 
remain a problem, the variation in completeness and 
quality of data is problematic. The use of the unwieldy 
Read codes system makes analysis difficult, and we 
observed significant differences across sites in reporting 
patterns. Some of these differences may be caused by 
the under-reporting of diagnostic variables, others by dif- 
ferences in coding approaches. However, the potential 
improvement in case finding, especially among patients 
with lower rates of utilisation in the preperiod, suggests 
that these barriers are worth confronting. Our develop- 
ment of new variables beyond those included in prior 
predictive modelling efforts 8 contributed substantially to 
enhanced case finding, and further work on variable 
development is likely to lead to further improvements. 
Again, these data are also useful in providing descriptive 
information on high-risk patients to help in intervention 
design (eg, documenting potential targets of opportun- 
ity such as uncontrolled hypertension or diabetes) . 

This study does not provide definitive findings on the 
value of developing individual site models compared to 
simply applying coefficients from multisite or national 
model coefficients to local data. Our four-site regression 
models generally had comparable PPVs to individual site 
models, but for the majority of sites the four-site regres- 
sion approach correctly identified a somewhat fewer 
number of patients with future admissions. Though it is 
tempting to speculate on whether differences in the 
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health needs of the population or coding differences 
affect model performance, we did not observe any clear 
patterns between the areas. Our analysis is somewhat 
limited by the small number of sites involved, which 
might cause somewhat greater variability in regression 
coefficients (regression coefficients for each of the five 
four-site models are available at http://www 
nuffieldtrust.org.uk/). Development of a national model 
using SUS data only is planned to further assess the 
need/value of locally developed models. 

Finally, it is worth noting that use of the GP registry 
data for the denominator also proved to be of significant 
importance. Many prior predictive modelling efforts 
have been limited to patients with utilisation history in 
whatever data sets were included. By including all 
patients in an area, not just those with prior use, the 
impact on predictive modelling of prior use was appar- 
ently enhanced. As a result, patients with more moder- 
ate levels of prior use and morbidity were found to be at 
higher risk than patients with no prior use at all, and 
were often assigned higher risk scores than when the 
analysis included just patients who had prior use. 
Accordingly, the use of the GP registry as the denomin- 
ator can improve rates of case finding and may permit 
identification of patients at earlier stages. 
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